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Amendments to the Claims; 

1 . (Previously amended) A computer-impiemeRted method of automaticaU y 
controlljnL c\CLUtion ot an appbt^Uion pro^ran b\ a ^.oi.ipu'oi >\stcm ihc app n.dtioii 
puuT..Mi hasmg a giaphtcai ti^ci nu^-itaco \o\ dss^ nu niidnndUv^r on <t J.spia^ touplod 
to the computer system conrptissnu; 

capturmg user uipui data ami images displayed by the giaplucal met iiuedace on the 
display during a recording phase of execution of tlie application program by the computer 
system; 

analyzing tlie captured user input data and displayed images to generate an execution 
scenario script dining the recording phase, wherein the captured input data is to be stored in a 
memory; 

generating simulated us^r itiput data based on the execution scenairio script during a 
playback phase of execution of the application program by the computer system and inputting 
the simulated user input data to the application program; 

perfoiming image analysis on images displayed by the graphical user interface on the 
display as a resuh of processing the simulated user input data during the playback phase and 
captured displayed images tVom the recording phase; 

checking recorded time conditions in the execution scenario script; to handle variations in 
playback speed, and 

automatically contiolltng execution of the application program by . the comg^^^^ 
based at least in part on the image analysis. 

2 . (Ortgina!) The method of claim 1 , wherein perfomiing image analysis comprises 
conipariiig the diapiayed images captured during the recording phase with displayed images from 
the phvvback phase. 

3 . (Original) The method of claim 1 , wherein tlie user input data compiises at least one 
of keyboard input data and mouse input data. 
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4. (Original) The method of claim 1 , wherein anal>'2:ing the captured user mpvtt data and 
displayed images in the recording phase comprises i dentifying actions of the application program 
corresponding to the captured user input data. 

5 (PiexKJUvsly <imended) Ihe metht»d of claun { , wheiem the eNecuUoii scenano ^cupi 
composes a script uicluding extended markup language (XML) tags. 

6. (Withdrawn) ITie method of claim I, xvherein analyzing the captured user input data 
and displayed ioiages during a recording phase comprises: 

detennining contours of objects shown in a displayed image; 
detecting an object of activity from among the objects; and 
detecting additional objects located adjacent to the object of activity. 

7. (Withdrawn) The method of claim \ , wherein performing image analysis comprises 
determining contours of objects in an image displayed by tlte graphical user interface dwing the 
playback phase and filtering the objects according to size to produce a set of hypotheses for 
active objects. 

8. (Wi thdrawn) The method of claim 7, wherein pertbrming image analysis ftu ther 
comprises filtering objects according to space layout to produce a first subset of hs^otheses for 
active objects. 

9. (Withdrawn) The method of claiot 7, wherein perfonning image analysis ftsrther 
comprises filtering objects by content to produce a second subset of hypotheses for acti ve 
objects. 

10. (Withdrawn) The metbod of claim 7, wherein perfomiing image analysis further 
comprises structural filtering of objects to produce a best hypothesis for an active object. 



3 qfll 



Altomm Dockeii^J P2M67 



PcmuAppim: 10/576,170 



I J . (W i thdrawn) The method of c l aim 7, w herein perforraing image mialysis ftixttjer 
comprises recalculating old actions for a new object identified as the active object. 

t t> m ;<-H H na L onipjiMniiL mstiucUon-- which xslien Lxccined ?esjli in auTtniiaatanv coniioliiii^ 
execution ot m appiicatu n piogram b\ a co^purei >%stcm, the appiicafion program having a 
gtaphical iisei niteitace for djvplayuig infoiniatjon on a display coupiedio the compute! svstem 
by 

captiiring user input data and images displayed by the gtaphical user interfe.ce interface 
on the display during a recording phase of execution of the application program by the computer 
system; 

analyzing tlie captured user input data and displayed images to generate an execution 
scenario script during the recording phase; 

geneniting simulated user input data based on the execution scenario script during a 
playbacic phase of execution of the application program by the computer system and inputting 
the simulated user input data to the application program; 

performing image analysis on images displayed by die graphical user interface on the 
display as a result of processing the simulated iiser input data during the playback phase and. 
captured displayed images troni the recording plmse; 

checking recorded time conditions in the execution scenario script to handle variations in 
playback speed, and 

automattcally ccntj'cning execution of the application program by the computer system 
based at least in part on the image analysis. 

13 (Cuticnih aniendeJ) The ameie conipntci seadablc uh dnnn oft, lium 12 uboK'ni 
uistiucuon^. ibi perfotnuug image anal>sis comprise insnuctions fot companiig ihe dispk>eJ 
images captured during the lecording phase widt displayed images from the playback phase. 

14. (Currently amended) The SfSiele computer readable medium of claim 1 2, wherein the 
user input data comprises at lea,st one of keyboard input data and mouse input data. 
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tx (Gurrentfy amemied) The mkie computer readable medium of chum 12 , wlierein 
instractions for anal ping tlie captujt^d itser inpiit data atid displayed images in the recording 
phase comprise instructions tor identifying actioris of the applieation pro-am coixespoadiag to 
the captured tjser iiipirt data. 

16. { Ciinentiv amended) The ffftiele computer feadalile ffle^ium of claim 12, wherein the 
execution: scenario scriptcotaprises a script including extendedmm'kup language (XML) tags. 

1 7. (Wididrawa) The article of daira 12, wherein instructions tbr analyzing the captured 
user input data and displayed images during a recording phase comprise instriictioits for: 

determining contours of objects shown in a displayed image- 
detecting an object of activity from among the objects; and 
detecting additionaJ objects located adjacent to the object of activity. 

18, (Withdrawn) The article of claim 12, wherein instructions for perfonning image 
analysis comprise instmctioiis for determining contours of objects in an ititage displayed by the 
graphical user interface during the playback phase and filtering the objects according to size to 
produce a set of hypotheses for active objects. 

19, (Withdrawn) The article of claim 1 8, wherein instmctions .for performing image 
analysis further comprise instructions for filtering objects according to space layout to produce a 
first subset of hypotheses for active objects. 

20 ( WithdurAii) The method ofclaun 1<S, whcrcm nistrui-tions Tot pciibinnng image 
analysis, luuhci comprsse insUucdorKs for fdieriug objects by content to produce a second subset 
of hypotheses for active objects. 
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2 i . (Withdrawn) The artkie of claim 18, wherein instoctions for perfbnning image 
analysis ftirther coraprise instructions for stmctaral filtering of objects to produce a best 
hypotliesis for an active object. 

22 (\\ uhdunvn> I he <«ticic ofciaim 18, wtieicm lasixuctjont, foj pwtt^unnig iindyc 
fimiK sib tui thei comprise instructions for recalculating old actions for a new object identified as 
the active object, 

23. (Previously amended) A cognitive control framework system for automatically 
controlling execution of an application program having a graphical user interface m a computer 
system comprising: 

a recording componerxt adapted to capture user input data and images displayed by Ihe 
graphical user mlerface dtjring a recofdixlg phase of execution of the application program, and to 
ajialyze the captured user input data and displayed images to generate an execution scenario 
script during the recording phase, 

a playback component adapted to generate simulated user input data based on the 
execution scenario script duiitig a playback phase of execution of the application program, to 
input the simulated user input data to the application program, to perform image analysis on 
images displayed by the graphical user interface as a result of processing the simulated user input 
data during the playback phase and captured displayed Images from the recording phase; 
checking recorded time conditions in the execution scenario script to handle vadations in 
playback speed, and to autcmaticaliy control execution of the application progmm by the 
computer system based at least in part on the image analysis, 

2 } (On^fiPalj The sv^tcni o! cld^m 23 \^hcicin the plnbatk v.on?|-n>neni in uiapud to 
compaie ihe (.hspldvcd images capluied dunng the {ecoulmg phai.e witti dibpid>cd imagv-s liom 
tlie playback phase. 

25. (Oiiginal) The system of claim 23, wherein the recording component is adapted to 
identify actions of the application program corresponding to the captured user input data. 
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26. ( Wi thdrawR) The system of claim 23, whereiii the recording compontmt is adapted to 
detemiine contours of objects sliowii in a displayed image, detect an object of acti vity front 
mtiong tfee objects, and detect additional objects located adjacent to the object of activity, 

27. { Withdiawn) "ilie system of claim 23, wherein the playback component is adapted to 
determine contours of objects in an image displayed by the gitiphical user interface during the 
playback phase and filter tlie objects according to size to produce a set of hypotheses for active 
objects, 

2S. ( Withdrawn) The system of claim 27, wherein the playback component is adapted to 
fiiter objects according to space layowt to produce a first subset of hypotheses for active objects. 

29. (Withdrawn) The system of claim 27, wherein the playback component is adapted to 
iiher objects by content to produce a second subset of h\^otheses for active objects. 

30. (Withdrami) The system of claim 27, wherein the playback component is adapted to 
structurally filter objects to produce a best hypothesis for an active object. 

3L -33- (Cancelkd) 
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